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ABSTRACT 



A method and system for modeling services available via a 
network include selecting a core service that is to be 
modeled, forming a discovery template that is specific to the 
selected core service, and automatically discovering the 
elements which cooperate to provide the core service. The 
discovery template includes instructions for implementing 
automated techniques for discovering service elements, and 
preferably services, which are anticipated as being coopera- 
tive in executing the core service. The systera includes a 
number of discovery modules for generating outputs indica- 
tive of the services and service elements. A discovery engine 
is responsive to the discovery template to invoke the mod- 
ules that are identified in the template as being relevant to 
discovering specified services and service elements. The 
template also identifies dependencies among the modules, so 
that the proper sequence of processing can be determined. In 
one embodiment, the discovery template is organized into 
sections, with each section (1) being specific to a type of 
service or service element, (2) specifying at least one 
discovery routine for identifying the specified type of ser- 
vice or service elements, and (3) specifying dependencies of 
the identified discovery routine on outputs of other discov- 
ery routines. Preferably, each section also includes instruc- 
tions for configuring the data that is output from the iden- 
tified discovery routine. 

21 Claims, 13 Drawing Sheets 
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AUTOMATED SERVICE ELEMENTS 
DISCOVERY USING CORE SERVICE 
SPECIFIC DISCOVERY TEMPLATES 

TECHNICAL FIELD 5 

The invention relates generally to methods and systems 
for discovering service elements that enable services via a 
network, as well as for discovering inter-relationships 
among the services. Such information may be used for 
constructing models of services, permitting network person- 10 
nel to assess the health of the service and to diagnose 
problems associated with the service. 

BACKGROUND ART 

15 

Originally, computer networks were designed to have a 
centralized, network topology. In such a topology, a cen- 
tralized mainframe computer is accessed by users at com- 
puter terminals via network connections. Applications and 
data are stored at the mainframe computer, but may be 2Q 
accessed by different users. However, a current trend in 
network design is to provide a topology that enables dis- 
tributed processing and peer-to-peer communications. 
Under this topology, network processing power is distrib- 
uted among a number of network sites that communicate on 25 
a peer-to-peer level. Often, there are a number of servers 
within the network and each server is accessible by a number 
of clients. Each server may be dedicated to a particular 
service, but this is not critical. Servers may communicate 
with one another in providing a service to a client. 30 

Networks vary significantly in scope. A local area net- 
work (LAN) is limited to network connectivity among 
computers that are in close proximity, typically less than one 
mile. A metropolitan area network (MAN) provides regional 
connectivity, such as within a major metropolitan area. A 35 
wide area network (WAN) links computers located in dif- 
ferent geographical areas, such as the computers of a cor- 
poration having business campuses in a number of different 
cities. A global area network (GAN) provides connectivity 
among computers in various nations. The most popular 40 
GAN is the network commonly referred to as the Internet. 

The decentralization of computer networks has increased 
the complexity of tracking network topology. The network 
components (i.e., "nodes") may be linked in any one of a 
variety of schemes. The nodes may include servers, hubs, 45 
routers, bridges, and the hardware for linking the various 
components. Systems for determining and graphically dis- 
playing the topology of a computer network are known. U.S. 
Pat. No. 5,276,789 to Besaw et al. and U.S. Pat. No. 
5,185,860 to Wu, both of which are assigned to the assignee 50 
of the present invention, describe such systems. As 
described in Besaw et al., the system retrieves a list of nodes 
and their interconnections from a database which can be 
manually built by a network administrator or automatically 
constructed using computer software. The system can be 55 
configured to provide any one of three views. An internet 
view shows nodes and interconnections of different net- 
works. A network view shows the nodes and interconnec- 
tions of a single network within the internet view. A segment 
view displays nodes connected within one segment of one of 60 
the networks. Selected nodes on the network, called discov- 
ery agents, can convey knowledge of the existence of other 
nodes. The network discovery system queries these discov- 
ery agents and obtains the information necessary to form a 
graphical display of the topology. The discovery agents can 65 
be periodically queried to determine if nodes have been 
added to the network. In a Transmission Controller Protocol/ 
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Internet Protocol (TCP/IP) network, the discovery agents are 
nodes that respond to queries for an address translation table 
which translates Internet Protocol (IP) addresses to physical 
addresses. 

The Besaw et al. and Wu systems operate well for 
graphically displaying hardware components and hardware 
connections within a network. From this information, a 
number of conclusions can be drawn regarding the present 
capabilities and future needs of the network. However, the 
inter-dependencies of the components in providing a par- 
ticular service are not apparent from the graphical display 
that is presented by the system. The complexities of such 
interdependencies continue to increase in all networks, par- 
ticularly the Internet. Moreover, these systems are designed 
in a monolithic manner. This does not allow the management 
system to be extended to discover and manage new service 
elements or new services. 

Another approach is described by J. L. Hellerstein in an 
article entitled "A Comparison of Techniques for Diagnos- 
ing Performance Problems in Information Systems: Case 
Study and Analytic Models," IBM Technical Report, 
September, 1994. Hellerstein proposes a measurement navi- 
gation graph (MNG) in which network measurements are 
represented by nodes and the relationships between the 
measurements are indicated by directed arcs. The relation- 
ships among measurements are used to diagnose problems. 
However, the approach has limitations, since MNGs only 
represent relationships among measurements. An ISP opera- 
tor must understand the details of the measurements (when, 
where, and how each measurement is performed) and their 
relationships to the different service elements. This under- 
standing is not readily available using the MNG approach. 
Furthermore, this publication does not explore ways of 
automatically discovering the relationships among measure- 
ments. 

The emergence of a variety of new services, such as 
World Wide Web (WWW) access, electronic commerce, 
multimedia conferencing, telecommuting, and virtual pri- 
vate network services, has contributed to the growing inter- 
est in network-based services. However, the increasing 
complexity of the services offered by a particular network 
causes a reduction in the number of experts having the 
domain knowledge necessary to diagnose and fix problems 
rapidly. Within the Internet, Internet Service Providers 
(ISft) offer their subscribers a number of complex services. 
An ISP must handle services that involve multiple complex 
relationships, not only among their service components 
(e.g., application servers, hosts, and network links), but also 
within other services. One example is the web service. This 
service will be described with reference to FIG. 1. Although 
it may appear to a subscriber of the ISP 10 that the web 
service is being exclusively provided by a web application 
server 12, there are various other services and service 
elements that contribute to the web service. For instance, to 
access the web server 12, a Domain Name Service (DNS) 
server 14 is accessed to provide the subscriber with the IP 
address of the web site. The access route includes one of the 
Points of Presence (POP) 16, a hub 18, and a router 20. Each 
POP houses modem banks, telco connections, and terminal 
servers. A subscriber request is forwarded to and handled by 
a web server application. The web page or pages being 
accessed may be stored on a back-end Network File System 
(NFS) 22, from which it is delivered to the web server on 
demand. When the subscriber perceives a degradation in the 
Quality of Service (QoS), the problem may be due to any of 
the web service components (e.g., the web application server 
12, the host machine on which the web application server is 
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executing, or the network links interconnecting the sub- template is combined with an engine for driving the discov- 

scriber to the web server), or may be due to the other ery routines, information is acquired for mapping depen- 

infrastructure services on which the web service depends dencies among actual services and actual service elements 

(e.g., DNS or NFS). The ISP system 10 of FIG. 1 is also within the network of interest. 

shown to include an authentication server 24 for performing 5 The term "service" as used herein is defined as "a func- 

a subscriber authentication service, a mail server 26 for tionality offered by a network to a user to perform a specific 

enabling email service (for login and email access), and set of tasks." Performance of the service involves a number 

front-end and back-end servers 28, 30 and 32 for allowing of cooperating elements, such as network routers, servers, 

Usenet access. applications and the like. Services include application ser- 

e , " , 4 , t lori ~ . . « . , in vices (such as web and email access) and network services 

Subscribers demand that ISPs offer reliable predictable 10 NetwQrks) 

services. To meet the expectations of subscribers and to jQe indudes a number of ^ modules for 

attract new subscribers, ISPs must measure and manage the executu / g me routines desi ^ to detect and ser- 

QoS of their service offerings. This requires a variety of vicc cltmtnts of thc network ^ system also includes thc 

tools that momtor and report on service-level metrics, such discovery engine for deploying the modules. The discovery 

as availability and performance of the services, and that 15 template is accessed by the engine to determine which.types 

provide health reports on the individual service components. Q f services and service elements are to be discovered and to 

Unfortunately, the majority of management systems have determine which modules should be deployed. Thus, the 

not kept pace with the service evolution. Available manage- discovery template includes identifications of multiple ser- 

ment systems lack the capability to capture and exploit the vice elements relevant to the core service and includes 

inter-relationships that exist among services available in a 20 identifications of discovery modules for detecting the actual 

network environment, such as the Internet. Typically, these services and service elements relevant to the core service. In 

management systems discover and manage service elements the preferred embodiment, the discovery template also 

in isolation. Moreover, these systems are implemented in a includes identifications of dependencies among the discov- 

monolithic manner and are not easily extensible to discov- fry modules. The discovery template may also include 

ering and managing new network services and service 25 instructions for configuring the instance information that is 

elements. Adding new discovery and management capabili- out P ut from the modules. 

ties to these systems requires extensive redesign and modi- The use of the discovery template to provide instance 

fication of the management system. information for modeling the core service preferably occurs 

. # * without human intervention. Since there are dependencies 

Each network is unique in various respects, such as the a the auto<liscovcry routin6S executed by & modules> 

configuration of servers, the types of application servers, the ^ ^ of 6X6CUtin the routines h , ically a coacem . 

service offerings, the organizational topology, and the inter- , Q onc Q to definin me d of discovery 

semce dependencies. Therefore, ui order to accurately the Ute ^ m ^ reflect ^ 

understand the operation of the network, specific models dependencies of the discovery modules . ^ is, a section is 

must be crafted for the services provided within the network. ^ onl after processing of all ^tions on which it 

However, handcrafting models of network services requires d ^ m ^ ach me , late the order u 

an enormous effort on the part of a human expert or group which disco h ^^0. In a second approach, the 

of everts. Furthermore new Internet services are mtio- sequencing is determined by the discovery engine. Again, 

duced at a rapid pace. Networking equipment with newer me disco t ^ ^ &vidcd into ^ but the 

capabiUtiesandfastero^raUonarcrjeingmtroducedrapidly fencing of ^ns is based upon identifications of 

and on an on-going basis. As networks and services evolve dependencies by the engine. Sections having no dependen- 

what is .needed is a system that is capable of d.scovenng and cies be iden(ified ^ ^ first . 

managing new services and service elements without requir- ^ can men ^ ^ ^iovs 

ing extensive redesign or modification. Since future services which have no , ^ and whicn have their depen . 

and service elements cannot be predicted with certainty, the dendes delermined b earlier processing . ^ * r epe ate d 

system must not be specific to the services and service umil aU of , he Qave ^ accessed ^ processed . 

elements it discovers and manages. m a third approach, the discovery modules cooperate to 

To manage services once they are discovered, what is determine the discovery sequencing. For example, a discov- 

needed is a method that accommodates the generation of a e ry module having a dependency on an output from a second 

model of a core service that is available via a network, with 5Q discovery module may register to receive the discovered 

the method being responsive to dependencies among ser- information from the second module. The dependent module 

vices and service elements which are relevant to providing ^ men notified when the necessary information is available, 

the core service. What is further needed is a system that while this adds complexity to the process, this third 

operates according to the method. approach allows the modules to execute in parallel. 

........nv „ c ixn™rnnM 55 In one embodiment, the network of interest is an Internet 

SUMMARY OF THE INVENTION ProvMer Qsp) system ^ discovery template may 

A method and system for discovering elements which be formed to be substantially generic to ISPs, but the 

cooperate in the performance of a core service among instance information received from the discovery modules in 

various services offered by a network include selecting the response to the discovery template is specific to the ISP of 

core service of interest and forming a discovery template 60 interest The instance information can then be used to 

that is specific to the selected core service. The discovery generate a graphical representation of interlinked nodes in 

template includes data relevant to triggering selected dis- which each node represents a service (e.g., mail service and 

covery routines for acquiring information that identifies authentication service) or a service element (e.g., host, 

services and service elements that are anticipated as being server and network link) and each interlink indicates an 

cooperative in execution of the core service. In the preferred 65 interdependency between two oodes. 

embodiment, the discovery template is specific to the core An advantage of the invention is that a discovery process 

service, but not specific to any network. When the discovery occurs automatically in its detection of services and service 
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elements that cooperate in executing a selected core service. 
The automated process for discovering services, service 
elements and dependencies may be executed periodically, so 
as to discover new services and service elements that may 
have been introduced into the network environment of 
interest. 

Another advantage is that the system is extensible to 
discover services or service elements that have been added 
to a network. When a new service or service element is 
introduced, a user can add a new discovery module for the 
service or service element and add the appropriate entries 
into the discovery template. The discovery engine is then 
able to discover instances of the new service or service 
element without any changes or enhancements to the dis- 
covery engine. This approach permits third party discovery 
modules to be easily and quickly introduced, without requir- 
ing any changes to the management system. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic view of an exemplary prior art 
Internet Service Provider (ISP) system. 

FIG. 2 is a schematic view of a process for modeling a 
selected service available via a network. 

FIG. 3 is a block diagram of components of a system for 
modeling a selected service available via a network, such as 
the ISP of FIG. 1. 

FIG. 4 is a block diagram of a management system having 
components of FIG. 3. 

FIG. 5 is an exemplary layout of components for utilizing 
the Read Mail service of the ISP of FIG. 1. 

FIG. 6 is a graph of nodes of a Read Mail service model 
configured using the system of FIG. 3. 

FIG. 7 is an alternative Read Mail service model graph. 

FIG. 8 is a schematic view of first phase internal discov- 
ery processing using the system of FIG. 3. 

FIG. 9 is a schematic view of second phase external 
discovery processing using the system of FIG. 3. 

FIG. 10 is a process flow of steps of the first phase 
discovery of FIG. 8. 

FIG. 11 is a process flow of steps of the second phase 
discovery of FIG. 9. 

FIG. 12 is a block diagram of the operation of the 
discovery engine of FIG. 4. 

FIG. 13 is a block diagram of the discovery engine of FIG. 
12. 

BEST MODE FOR CARRYING OUT THE 
INVENTION 

The invention relates to the discovery process for 
enabling automated detection of service elements and/or 
services that are utilized by a specific network to provide a 
particular service. One application of the invention is to 
model the particular service, so that dependencies among 
services and service elements (e.g., servers, hosts and net- 
work links) may be readily determined by network person- 
nel. The discovery process utilizes a service-specific discov- 
ery template as a primary means to provide extensibility of 
an auto-discovery system, since the template drives the 
performance of the process. Optionally, a service model 
template may be used in combination with the discovery 
template to form an instance of the core service offered over 
the network. However, a service model template is not 
critical to the practice of the auto -discovery process. It is 
also noted that while the invention will be described prima- 
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rily with reference to application to an ISP system, the 
process has other applications. That is, the process may be 
used to provide a service model for other providers of 
services in network environments (e.g., for services within 

5 a corporation's network). 

FIG. 2 is an overview of the auto-discovery process for 
generating a service model instance 200 for a particular 
network 202. In FIG, 2, the rectangles represent data stores 
and the ellipses represent processing elements. A service- 

10 specific discovery template 204 is accessed upon initiation 
of the auto-discovery process. The discovery template 
defines the services and service elements that are anticipated 
as cooperating to provide the core service. Moreover, the 
discovery template identifies particular discovery tools (i.e., 

15 discovery modules and/or agents) that are to be used for each 
service and service element, as well as any dependencies 
that a particular discovery tool may have on outputs from 
other discovery tools. The template preferably includes 
instructions for configuring the outputs of the discovery 

20 tools. A more thorough description of the discovery template 
will follow, particularly with reference to Table 2, which is 
an example of a discovery template specification. A more 
thorough description of the discovery tools 206 will also be 
provided below, with reference to FIGS. 12 and 13. 

25 The auto-discovery approach is dividable into two phases. 
In the first phase auto-discovery procedure 208, the infor- 
mation that is available in the domain name system of an ISP 
is used to discover the existence and the relationships among 
different services. In order to allow subscribers to access a 

30 host using its host name, rather than requiring specification 
of an IP address that is more difficult to remember, DNS is 
used to map the host names of the IP addresses for all hosts 
in the ISP system. Moreover, the exchange of email mes- 
sages across hosts occurs using the mail exchange records 

35 (MX records) maintained by the DNS. In summary, the 
domain name system stores a wealth of information that is 
used in the first phase of the auto-discovery process to 
discover Internet services and relationships among the ser- 
vices. Other mechanisms may be used in this first phase to 

40 supplement the information acquired from the domain name 
system. 

Instance information 210 is acquired in the first phase 
procedure 208. In addition to the identification of services 
and service elements, the instance information 210 identifies 

45 dependencies. There are a number of inter-dependencies of 
concern to the auto -discovery process. These inter- 
dependencies include execution dependencies, component 
dependencies, inter-service dependencies, and organiza- 
tional dependencies. An execution dependency relates 

50 directly to an application server process being executed on 
a host machine. The types of application servers that are 
executed on host machines include web, email, news, DNS 
and NFS. A component dependency occurs in order to 
ensure scalability and redundancy of a service. For example, 

55 a web service may be provided collectively by a number of 
"front-end servers" (FESs), with round-robin DNS sched- 
uling being used to provide subscribers with a domain name 
that maps to one of the FESs. ISPs often replicate web, email 
and news content across a number of servers. The round - 

60 robin scheduling balances the load among the servers. The 
servers are grouped together and assigned a single domain 
name in the DNS database. When the DNS server receives 
a request for the domain name, the IP address of one of the 
servers is acquired in the round-robin scheme. 

65 An inter-service dependency occurs when one service 
accesses another service for its proper operation. For 
example, a web service depends on a DNS service to allow 
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the subscriber to connect the web server host using its IP 
address, and an NFS service is used to access the web 
content. As another example, a Read Mail service depends 
on an authentication service to verify the identity of the 
subscriber, uses a DNS service to log the subscriber's IP 5 
address, and uses an NFS service to access the mail content. 

Finally, an organization dependency occurs when there 
are different ISP operations personnel (e.g., subject matter 
experts) who are responsible for different services and 
service elements. For example, an ISP may have a first 1Q 
supervisor managing the web service, a second supervisor 
managing DNS, and a third supervisor managing NFS. 
Operational responsibilities may be delegated based upon 
the geographical location of the service elements. Since the 
precise organization structure may vary from one ISP to 
another, the auto -discovery mechanism provides a means by 15 
which it can be quickly customized for a particular ISP 
system. 

The first phase auto-discovery procedure 208 provides 
discovered instance information 210 that identifies most of 
the execution, component and organization dependencies, as 20 
well as some of the inter-service dependencies. A partial 
service model instance generator 212 is then used to provide 
a partial service model instance 214. Optionally, configura- 
tion data 216 may be used to allow an operator to customize 
a service model instance 200. Using a configuration inter- 25 
face which will be described with reference to FIG. 4, the 
operator can specify categorization criteria for services and 
service elements. Thus, the service model instance 200 can 
be configured to meet the specific needs of the operator. 

In a second phase auto-discovery procedure 218, an 30 
internal view of the network 202 is acquired. The internal 
discovery of the second phase is intended to fill any "holes" 
in the partial service model 214, and particularly focuses on 
identifying inter-service dependencies. There are two basic 
approaches to the internal discovery of the second phase. 35 
One approach is the use of network probes, which are 
implemented in software and are installed at strategic loca- 
tions on the network to acquire information from headers of 
packet transmissions. By processing the headers of packets, 
a software probe can deduce many of the relationships that 40 
exist among servers. The second basic approach is to use 
special-purpose discovery agents that are installed on the 
ISP hosts to discover relationships among services. Rather 
than examining headers of packet transmissions, the special- 
purpose agents use a number of operating systems and 45 
application-specific mechanisms (such as processing service 
configuration information and application -dependent moni- 
toring tables) to discover inter-service dependencies. 

The inter-service dependency information 220 from the 
second phase auto -discovery procedure 218 is combined 50 
with the partial service model instance 214 using a second 
service model instance generator 222. The output of the 
second service model instance generator is the completed 
service model instance 200. 

The invention will be described primarily in its preferred 55 
embodiment of using the discovery template to detect both 
services and service elements to form a service model. 
However, a discovery template in accordance with the 
invention may be restricted to detecting service elements 
and the discovered instance information may be utilized for 60 
purposes other than forming a model of the type to be 
described below. 

ONE EMBODIMENT FOR USING THE 

INSTANCE INFORMATION 65 

As previously noted, the use of the discovery template 
204 is optionally combined with a service model template in 



136 Bl 

8 

the process of generating the service model instance 200. An 
overview of the template-driven procedure is illustrated in 
FIG. 3. The service model template is a generic specification 
of the service topology and measurement topology for the 
service of interest (e.g., Read Mail service). Depending on 
the service being modeled and the service elements that are 
likely to be involved, the template defines nodes of various 
types (e.g., hosts, servers, network finks, and services) and 
their associated measurements. Moreover, the template indi- 
cates the dependencies among the nodes, such as the depen- 
dency of the service on other services (e.g., the Read Mail 
service which refers to a subscriber accessing his/her mail- 
box depends on the authentication and NFS services). In the 
preferred embodiment, the template also includes default 
state computation rules for specified nodes, so that the state 
(i.e., "health") of a node can be computed based upon 
measurements associated with the node and upon states of 
dependencies of the node. 

The service model template 34 is not specific to any JSP 
system in the sense that it does not specifically reference any 
hosts, servers, network links, service groups, or other ser- 
vices or service elements in the ISP system. The template 
may be considered as a "lifeless tree.'* There is no associa- 
tion between nodes in the service model template and 
running elements, such as hosts and servers. However, the 
information contained in the service template for a node may 
include the element type, the element dependencies, the 
measurement definitions (e.g., agent to run, the format of the 
run string, the number and type of parameters, and the 
format of the output), default state computation rules, 
default thresholds, default baselines, and default alarm gen- 
eration and correlation rules. 

In database terminology, the service model template 34 is 
the schema. On the other hand, an instance defines the 
records in the database and the values of the static informa- 
tion. In FIG. 3, the discovered instance 36 is determined 
using auto-discovery, as will be explained fully below. 
Information regarding the services and service elements 
(e.g., servers, hosts and links) that exist in the ISP system or 
other service provider systems may be auto-discovered. The 
store representing auto-discovered information shall be 
referred to as the auto-discovered instance 36. 

The service model creation engine 38 of FIG. 3 is used to 
generate a service model instance 40 based on the service 
model template 34 and auto-discovered instance 36. Ideally, 
all of the discovered information is available prior to instan- 
tiation. However, in many cases, discovery has to be per- 
formed in multiple phases, such as the two outlined above. 
In such cases, instantiation may occur partially after each 
phase of discovery. The main advantage of providing a 
partial service model instance is that the partially completed 
service model can provide a guide to identify the additional 
information needed in subsequent phases of discovery. The 
service model creation engine 38 encompasses the functions 
of the partial service model instance generator 212 and the 
second service model instance generator 222 of FIG. 2. 
Since new elements and services may be added to the ISP 
system over time, the service model instantiation process 
has to be repeated periodically to augment the initially 
created service model instance. 

Unlike the service model template 34, the service model 
instance 40 is specific to an ISP system. The process of 
constructing the service model instance using the service 
model template and the auto-discovered instance 36 is 
referred to as the "instantiation*' of the service model. As 
previously noted, the relationship between the service model 
template and the auto-discovered instance is analogous to 
the relationship between the schema and records in a data- 
base. 
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The service model instance 40 maps services and service model instance 40 of FIG. 3. As previously noted, a require- 

elements that exist in a particular ISP system with nodes in ment of the instantiation of a service model is the generation 

the service model template. In doing so, the service model 0 f the service model template for the particular service or 

instance also specifies a set of measurements that must be services of interest. The template may be handcrafted, 

rSifS^S? the services and semce elements 10 the 5 ? referabl y b y a domain ex P ert of the service - 1136 template 

S ^!L C . S ^ St£m /,. * JL . includes a specification of the elements involved in provid- 

The service model instance 40 may be used by a view . , , , . . , 

generator 42. In the preferred embodiment, the service ^ th « service such as servers and network links and he 

model instance is represented as a graph of the nodes and dependencies of the service on other services, such as the 

edges that identify dependencies of the nodes. Different dependency of Read Mail service on the authentication and 

management functions of an ISP will benefit from viewing NFS services. As will be explained more fully below, the 

different subsets of nodes and edges of the service model discovery process includes designation of a discovery tem- 

instance 40. Thus, the view generator 42 may be used by plate 48. The discovery template specifies the types of 

operations personnel to provide an "operations view" of services and the service elements to be discovered. The 

only the services and service elements that fall in a particular discovery template also includes specifications of discovery 

domain of control of the personnel On the other hand, a modules 50 52 and 54 to be m the discovery of the 

"customer support view* may provide an end-to-end view dfied and ^ m&nXs 

for the customer support personnel of the ISP. A planning r 

view** may be used to graphically present information about The service model template 34, the discovery template 48, 

usage and performance trends, so that future requirements and the discovery modules 50, 52 and 54 are utilized within 

can be ascertained. 20 a mana g e ment system 56 for forming the service model 

Even when different management functions are interested instance 40. Another key component of the management 

in the same subset of nodes and edges of the service model m fa ft ^ me 58 that proccsses information 

instance 40, there may be different interest with regard to ^ ^ ^ fc 4g ^ ^ 

rules to be used for state computations. For instances, a . • » - I _l- j % en *a 

management application that visually depicts a color-coded 25 ^ kcs thc appropnate mscovery modules 50-54, 

hierarchical representation of the service model instance to interprets the outputs of the invoked modules and stores the 

operations personnel may require that hierarchical state 0Ut P uts m memorv ( or m a database) as the discovered 

propagation rules be employed in the service model instance 36, which is made accessible to the service model 

instance. In this manner, viewing the state of the top-level creation engine 38. The discovery engine 58 supports a 

nodes of the service model, an ISP operator can determine 30 configuration interface 60 that allows ISP operations per- 

whether any problem has been detected in the ISP system. In sonnel to control and customize the discovery process, 

contrast, an application that is responsible for automatically Through the configuration interface 60, an ISP operator can 

detecting, diagnosing and reporting the root-causes of prob- restrict the discovery to specified IP address ranges or host 

lems in the ISP system may prefer not to use a hierarchical name patterns. Furthermore, the operator can specify nam- 

state propagation rule, since it is interested in the state of 35 ing conventions used by the ISP (e.g., for naming terminal 

each node of the service model independently of the state of servers and POP sites), which are used by some of the 

other nodes. discovery modules 50-54. 

The view generator 42 may be used to define the subset m „ 

of nodes of a service model instance that are of interest to a ™ e configuration interface 60 serves as a way for an ISP 

management application, as well as the method for evalu- « op^or to quickly customize the service model instance 40 

ating the states of the nodes of interest. The measurements 15 generated by the process. Using the configuration 

that are associated with the nodes in the service model interface, the operator can also specify categorization cnte- 

instance are common across the different views. However, ria for services and service elements. For instance, all the 

the rules that apply to computing the state of a particular m ail services could fall in one category, while the DNS 

node based upon relevant measurements and the states of 45 servers could fall in another. The categories assigned to 

dependent nodes may be different for the different views. services and service elements can represent the organiza- 

A measurement agent configurator 44 may be used to tional st ™ cture °f the ISP ' geographical location of the 

extract information from the service model instance 40, offenn S the *™ ( e -S- servers in San Francisco 

when the information is relevant to scheduling tests and faU L m ° oe c ^ 0 ^ m New Y 0 ' k faU m 

retrieving test results. At the completion of the service model 50 another) or differences in business functions of the service 

instance 40, static configuration information as to how the < e *- web that 15 on behalf of business 

tests of various elements are to be run may be merged with customers, as opposed to local web servers that the ISP 

default algorithm descriptions into a service model instance offers for Priding access to the d ia l-up subscribers), 

file. State computation rules, thresholds and alarm correla- i n the preferred embodiment, the configuration interface 

tion rules defaults may be overridden using the measurement 55 60 is implemented using a graphical user interface (GUI) at 

agent configurator. Specific measurements may be enabled an operator computing station. An example of a configura- 

or disabled. A final measurement agent specification 46 is tion specification that is entered using the interface 60 is 

generated by the configurator for the specific ISP. The shown in Table 1. 
measurements that are identified in the specification 46 are 

executed using software agents 45. The results of the mea- 60 TABLE 1 
surements are analyzed and used by a service model man- 



ager 47 to compute the states of different nodes in the service Ho8ts ¥ 10.1.204.1-10.1.205.1 

j i ^exclude all hosts with IP addresses in this range. These hosts represent 

moaeL # subscriber home PCs 



OVERVIEW OF INSTANTIATION WebServers category name *. com.faketsp.net WebHost 

fi 5 #servers with names of the form *.com. fakeisp.net are Web hosting 

FIG. 4 represents an overview of the process for gener- servers 
ating the discovered instance that is used to form the service 
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TABLE 1-continued TABLE 2-continued 



TerminalServcrs extract name max(0-9]{3}<PopSite>[0-9]t. discovery-outputs=ipAddress, hostName, state, category 

fakeisp.net discovery- ins tanceKeyo<ipAddress>:Host 

^Terminal servers must match the naming pattern above. Extract the POP 5 discovery-dependencies-ExternalNameServers 
# site name from the terminal server's name [SMTP Servers] 



— ,i — — . ii - i n i - — — discovery- modules Disco verSmtpServersxlass 

discovery-arguments— url http://Um-pc2/scripts/discEngirieAPLpl 

The discovery modules 50-54 obtain the configuration dKcovery-ouipu^^ 

. . .. _„ , disco ve ry- ins ta nee Ke y= <ip Address > :S mtp __Ma ii 

cntena from the discovery engine 58. The modules execute 10 discovery-dependencies-Hosts^xternalNameServers 

the appropriate discovery techniques, and as part of their [POP3Servcrs] 

outputs, record the categories to which the different services discovery-moduie=DiscoverPop3Servers.ciass 

and service elements belong. The category information is discovery-arguments-uri http^^ci/so^mc^eAPi^ 

_ _ i , , discovcry-outpuls=ipAddress, hostName, relatcdSmtpServcr, category 

stored at the discovery instance 36 and is interpreted by the discovery-ir*tanceKey=<ipAddress>:Pcp3_Maii 

Service model creation engine 38 in a manner that is re flee- 15 discovery-dependencies-Hosts^xternalNameServers 

tive of the service model template 34. [HTTPServers] 

discovery- module= Discover HttpServers.class 

The following sections provide details regarding imple- discovery-arguments— url http://ism-pc2/scripts/discEngineA* > I.pl 

mentation of the process described with reference to FIGS. discovery-outputs*=ipAddrcss, hostName, serverType, category 

2 and 3. Since the service models template 34 depends on the discovery-uwtanceKey- «pAddie»>ttfeb 

. .1 ,• discovery-dependencies-Hosts 

syntax specified m the discovery template 48, the discovery 20 [^55^^] 

template specification is considered first. All the templates discovery-module=DiscoverNFSServers.dass 

and instances presented as examples use the INI file format discovery-arguments— url htt|j://Um-pc2/scTir^/discEngineAPLpl 

that is commonly used to represent the content of system d^very-outpuis^ P Addre« hostNam^catcgory 

„, . w . r , , , , J discov e r y- ins ta nee Key^ <ip Address > :NFS 

files in Microsoft Windows-based personal computer sys- discovery-depenendencies-Hosts 

tems. However, the process may be implemented using other 25 [Host-Groups] 

specification and schema definition technologies (e.g., the discovery-module=DiscoverHostGroups.class 

Common Information Model specified by the Desktop Man- d^eryHugmneoto^^^ 

_ , „ 1 Ivn r , * 1/ discovery-outputs—groupName, groupComponentsUst, category 

agement Task Force). Per the INI format, a template or an discovery- instanceKey^xgroupNamoiHostGroup . 

instance is Organized as different Sections, each of which is discovery-dependencies=ExternalNameServers . 

specified by a unique stream enclosed within a pair of square 30 [WebService Groups] 

brackets ("[<Section name>]"). hostN^e^nailfcsZl.fakeisp.net 

N u ' relatedSmtpServer=smtp.fakeisp. net 

category- 

DISCOVERY TEMPLATE SPECIFICATION [io.i37.i96.54:Po P 3„Maii] 

ipAddress=10.137.196.54 

An example discovery template specification is shown in 35 hostName-mailfes22.fakeisp.net 

Table 2. Each section of the discovery template defines a ^dSmt P Serv C r^mt P .fakei 5p .net 

specific service or service element. The first four sections [io.137.196.56:Po P 3 Mail] 

represent templates for discovery of external name servers, ipAddrcss«io.i37.i96.56 

hosts, Mail SMTP servers, and Matt POP3 servers. The hostName-maiifes23.fakeisp.net 

"module" variable specification in each section identifies the 40 ^tedSmtpServer-smtp.fakeisp.net 

r . category— 

discovery module 50-54 of FIG. 4 that is to be used for the [io.i37.i9G.58:Web] 

particular service or service element. The "arguments'* vari- ipAddress-io.i 37.196.58 

able represents arguments that are to be passed by the hostName-www.fakeisp.net 

discovery engine 58 to the discovery module during deploy- ^^^^^ 

ment. The "outputs'* variable defines the number and names 45 [10. 137.196. 69 :Wcb] 

of the columns in the output of the discovery module. The ipAddress=10.137.l96.69 

"instanceKey" variable denotes the index that is to be used hostName=www.xyz.com.fakeisp.net 

to access the discovery instance 36 corresponding to each Stego^WetSost^ 1 2 

row of output generated by the discovery module . The name [io.i37.i96.70:Web] 

or names specified within angled brackets (<>) on the right 50 ipAddress-io.i37.i96.70 

side of the instanceKey assignment must correspond to one hostName=www.abc.com.fakeisp.net 

of the column names specified in the output assignment. Mtego^WetSosT' 1 2 

Finally, the "dependencies** variable indicates the dependen- [10.1 74.173.23 :NFS1 

cies that a discovery module has on other modules. The ipAddress-io.i74.i73.23 

discovery engine may use this information to select a 55 hostName«*dnsi .fateisp.net 

sequence in which it processes the discovery templates. [pop3.fakeisp.net:HostGroup] 

groupNamc=pop3.fakcisp .net 

groupComponenUList-10. 137. 196.52:10.137.196.54:10.1 37.196.56 

TABLE 2 • [rxjp3.fakeisp.net:Pop3_MailServiceGroup] 

j. j , serviceGrpName=pop3. fake isp.net 

[HxternalNameServersJ 6Q ^ o ^ Q) _ nc ^ij^ 10>1 3 7il 9 6 .52:io.l37.196.54:10.137.396J6 

discovery-modu]e=DiscoverExteraaINameServere .class ** r 1 

discovery-arguments=-url http ://ism-pc2/scripU/discEngineAPI.pl [ExternalDnsServer Category J 

discovery-outputs-ipAddress, hostName, domainName, category categoryName-ExtcrnalDnsServcr 

discovery-instanceKcy=<ipAddress>:DNS [Internal Webserver :Category] 

discovery-dependencies- categoryName-InternalWeb 

[Hosts] [WebHostedServertCategory] 

discovery-module»DiscoverHosts. class 65 categoryName-WebHost 

discovery-arguments=-url http://ism-pc2/scripts/discEngineAPI.pl - — .. - ■ ■ ■ — , ■■ „ . . . . ■ . , - ■ „ -, -■ - 
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Because the ISP environment has a heterogeneous set of 
elements (e.g., host nodes, routers, application servers, ser- 
vices and inter-service dependencies), sections of the dis- 
covery template must be processed in an order in which 
elements having no dependencies are considered first. Sec- 
tions have dependencies can be processed after the depen- 
dencies have been completed. There are at least three 
approaches to ordering the processing of sections. One 
approach is to order the sections in the template to reflect the 
dependencies among sections. Thus, a section appears in the 
template only after the appearance of all sections on which 
it depends. The discovery engine 58 can then process the 
sections in order. Another approach is to allow the discovery 
engine to dictate the sequence of discovery. By considering 
the values of the dependencies variable within the sections 
of the discovery template, the discovery engine can deter- 
mine the order in which the sections must be processed. 
Sections of the template having no dependencies are pro- 
cessed first. After all such sections are processed, the dis- 
covery engine iterates through the list of template sections, 
choosing sections which have not been processed and which 
have their dependencies determined by earlier processing. 
This procedure is repeated until all of the sections have been 
processed. In the third approach, the sequencing is driven by 
the discovery modules 50-54 themselves. In this 
embodiment, the discovery engine processes the template 
once, invoking the discovery modules simultaneously. The 
discovery modules determine when the different elements of 
the ISP system are discovered. When a new instance is 
detected by a discovery module, it forwards the results to the 
discovery engine. Based on the dependencies on the discov- 
ery module, as specified in the discovery template, the 
engine forwards the results to other discovery modules for 
which the results are relevant. The availability of the new 
results may trigger discovery by other modules. This pro- 
cedure is repeated until all of the sections have been fully 
processed. In an alternative implementation, the discovery 
modules can communicate with one another without involv- 
ing the discovery engine. 

DISCOVERED INSTANCE SPECIFICATION 

Table 3 is a portion of an example of the discovered 
instance 36 of FIG. 4. Section names in the discovered 
instance correspond to the instanceKey variable specifica- 
tions and the discovery template of Table 2. For each of the 
output variables in the discovery template, there is an 
assignment in the discovered instance of Table 3. The ISP 
system being discovered in this example is assumed to have 
the domain name fakeisp.net. 

TABLE 3 

[10.174.173.23:DNS] 
ipAddress=10.1 74.173.23 
hostName-dnsl.fakeisp.net 
domainName-fakcisp. net 
category-ExternalDnsServer 
[10.174.173.23:Host] 
ipAddress=10.1 74.173.23 
hostName=dnsl .fakeisp.net 
state- Alive 
category- 

[10.137.196.52:Host] 
ipAddress-10.1 37.196.52 

hostName=mailfes21 . fakeisp . net :smtp. fakeisp .net 

state-Alive 

category- 

(10.137.196.54:Host] 
ipAddress=10.137.196.54 



TABLE 3-continued 



hostNarne=mailfes22. fake Lsp . net 

state* Alive 
5 category- 

[10.137.196.56:Hostl 

ipAddress-10.137.196.56 

bostName-mail fes23,fakeisp . net 

state= Alive 

category** 
10 [10.137.196.58:Host] 

ipAddress~10.mi96.58 

hostName=»www. fakeisp.net 

state- Alive 

category- 

[10.137.196.69:Host] 
15 ipAddress-10.137.196.69 

hostName-www.xyz.com. fakeisp.net 

state-Alive 

category- 

[10.137.196.70:Host] 
ipAddress=10.137. 196.70 
_ hostName-www.abc.com.fakeisp.net 
state- Alive 
category- 

[iai37.196.52:Smtp_Mail] 
ipAddrcss-10.137.196.52 
hostName=mailfes2 1 .fakeisp.net :s mtp.fakeisp. net 
category-Ex ternalSmtpServer 
25 [10.137.196.52:Pop3_Mail] 
ipAddxess=10.137.196.52 
hostName-mailfes21.fakeisp.net 
ielatedSmtpServe r-smtp.fakeisp.net 
category- 

[10.137.196.54:Pop3_MaU] 
30 ipAddress-10.137.196.54 

hostName-mailfes22.fakeisp.net 

relatedSmtpServer-smtp.fakeisp.net 

category- 

[10.137.196.56:Pop3_MaU] 
ipAddress~10.137.196.56 
35 hostName-mailfes23.fakeisp.net 

rclatedSmtpServer-smtp.fakeisp.net 
category- 



Next, a portion of a service model template specification is 

40 presented in Table 4 as an example service model template 
34. The service model template contains the intelligence to 
map the discovered instance into the service model nodes. 
The discovery modules (and hence the discovered instance) 
are designed independently of the service model instance 

45 being generated. This enables the same discovered instance 
to be used in the generation of different service models (e.g., 
for different services). Each section in the service model 
template represents a type of node in the service model 
instance 40 and contains a series of instructions for creating 

50 a node in the service model instance. The service model 
creation engine 38 processes sections of the service model 
template, one at a time, attempting to match the template 
with elements in the discovered instance 36. Each element 
in the discovered instance corresponds to a section of the 

55 discovered instance specification. Lines beginning with a 
represent comments. These lines are ignored by the service 
model creation engine 38 when it processes the service 
model template. 

60 TABLE 4 



[SM-Host] 

;Host Node in the service model 
match-f-VHost] 
;for each discovered Host 
65 instancckcy-<ipAddress>:SM-H05t 

measuiements=TCP-Connection Rate(<ipAddress>), 
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TABLE 4-continued 



TABLE 4-continued 



VMstat(<ipAddress>),[Fstat(<ipAddress>) 
;these are measurements of the host 
;next copy its attributes to the SM node 
hostname=<hostName> 
ip Address- <ap Address> 
state— cstate> 
category- <catcgory> 
[SM-Web] 

;SM node for a web server 
match»(*:Web) 

;match all discovered Web server instances 
instanceKey-<ipAddress>:SM-Web 

co mponent5=<rpAddrcss > :SM-Host, <ipAddress>-msl P :SM-link 

measuremeats=HTTP-TCPConnection Tune(<ipAddress>) 

;next copy all the attributes from the server discovered instance 

stateComputation Rule=measuiementsOnly 

serverType=<serverType> 

hostName-<hostName> 

ipAddrcsss<dpAddress> 

category=append(<category>, <ipAddress>:SM-Host?<category>) 

[SM-WebService] 

;a web service node 

match=|*:SM-Web] 

;there is a web service node corresponding to each web server node 
instanceKey- <ip Address>: SM- WebService 
measmement^HnT- Avail ability (<ip Address^, HTTP-Tbtal 

ResponseTime(<ipAddress>), HTTP-DnsTime (<ipAddress>) 
componcnts»<ipAddrcss>:SM-Web, <<pAddress>,Web>:SM-NFS, 

«ipAddress>, Web>:SM-DNS 
;NFS and DNS service dependencies will be determined by phase 2 
discovery category— <category> 
[SM-WebServiceGroup] 
rnatch=[*:WebServiceGroup] 

instanceKey- <serviceGrp Name> :SM- WebServiceGroup 

co mponcat5=list(<scrviccGrp Components U5t>):SM- WcbService 

category=<category> 

category^ppend(list(<service<jTpComponentslist>):SM-WebService? 

<category>) 

[SM-TbpLevel-Web] 

instanceKey-Web:SM-TopLevelWeb 

componcnts=* :SM- WebService, * :SM- WebServiceGroup 

jcomponents are all web services and all web service groups 

[SM-DNS] 

match={*:DNS] 

inst£nceKey=<ipAddress>:SM-DNS 
components-<rpAddjess>-msIP:SM-Ijnk 
measuremeats=DNS-Availability(<ipAddrcss>), 
DNS-CacheHitResponseTime(<ipAddress>) 
stateComputationRule-measurementsOnly 
ip Add rcss=cip Addrcss> 
hos tName=<hostName> 
domainName-<domainName> 
category-<catcgory> 
[SM-NFS] 
match=(*:NFS] 

instanceKey- <ipAddress:>:SM-NFS 
components=<ipAddiess>:SM-Host 

measurements*NFS-TotalCalls(<ipAddress>) ) NFS-DupReqs 
(<ipAddress>), NFS-HmeOutPercent(<ipAddress>) 

stateComputa tionRule-measurementsOnl y 

ip Add ressa cip Address> 

hos tName-<hostName> 

[SM-Pop3_Mail] 

match={ *:Pop3_Mail] 

instanceKey- <ipAddress>:SM-POP3_Mail 

components»<ipAddiess>:SM-Host 

meamrements=POP3-TCPC^nnecUonTune(<ipAddress>) 

hos tName- <hostName> 

category=<category> 

relate dSmtpServer= <relat£dSmtpSe rver> 

ip Add ress- <ip Address> 

[SM-RcadMailService] 

match=( *:SM-Pop3_Mail] 

instanceKey- <ipAddress> ;SM-ReadMailService 

measurements=POP3-Avaiiability (<ipAddiess>) > 
POPS-TctomesrxjnseTirneC^pAddress^, 
POP3-AuthenticatfonTirne (<ipAddress>) 

stateComputationRule-default 

components»<ipAddiess>:SM-Pop3_Mail, «ipAddress>4 > op3__Mail: 



SM-NFS>, «ipAddress>^ > op3_Mail:SM-Auth> 
catego ry=<category> 
5 [SM-ReadMaaServiceGroup] 
match=[ * :Pop3MailServiceGroup] 

instanceKey- <serviceGrpName> ;SM- Read Mail ServiceGroup 
components-listC<serviceGrpComponentsIist>) :SM-ReadMaOService 
category=<catcgory> 
[SM-TopLevel-ReadMail] 
10 instanceKey-ReadMail:SM-TbpLevel-ReadMail 

components- * :SM-ReadMailScrvice, * ;SM-ReadMailScrviceGroup 

[SM-Category] 

match-[ * :Category] 

instanceKcy«<:category Name> :SM-Category 
components- * :SM- * ?category=<categoryName> 
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Most sections of the service model template 34 begin with 
a "match" criteria. The match criteria for a section of the 
service model template specifies the discovery instances that 
are relevant to the section under consideration. For instance, 

20 the match criteria corresponding to the host node's specifi- 
cation (SM-Host) in the discovery template indicates that the 
corresponding discovered instances are those of type Host. 
The match criteria is specified as a regular expression that is 
matched against section names (instance keys) of the dis- 

25 covered instance 36. For each object (section) in the dis- 
covered instance that matches the regular expression speci- 
fied in the match criteria, a corresponding node is 
instantiated in the service model instance 40. 

Each section of the service model template 34 can match 

30 discovered instances of at least one type. When a discovered 
instance satisfies the match criteria specified in a section of 
the service model template 34, any of the attributes of the 
discovered instance can be referred to in the subsequent 
instructions of the service model template's section. The 
absence of a match criteria in the specification of a section 

35 of the service model template indicates that there is only one 
instance of that type for the particular ISP. 

The instanceKey variable in Table 4 denotes the key that 
is to be used to refer to a service model node that is 
instantiated by the section of the template under consider- 

40 ation. The attributes enclosed within the angled brackets 
("<>") must be one of the attributes of the elements of the 
discovered instance for which there is a reference by the 
match criteria. 
The "components" instruction specifies the parent-child 

45 relationship in a service model instance. Various types of 
dependencies (i.e., execution dependencies, component 
dependencies, inter-service dependencies and organizational 
dependencies) are captured by this specification. The com- 
ponents list specified must make reference to the node in the 

50 service model instance 40 that is to be generated from the 
service model template 34. The components list refers to 
specific nodes, all nodes of a specific type, and all nodes of 
a different specific type that have a specific attribute value. 
Sections of the service model template that refer to leaf 

55 nodes of the service model instance do not have component 
specifications. 

The "measurements" instruction specifies a list of mea- 
surements that must be targeted at the corresponding node in 
the service mode instance 40. By processing the measure- 

60 ment specifications of nodes in the service model instance, 
the measurement agent configurator 44 of FIG. 3 can deter- 
mine the agents that must be scheduled for execution against 
each element of the discovered instance 36. It should be 
noted that not all nodes in the service model instance have 

65 measurement specifications. 

The "StateComputationRule" instruction covers how the 
states of the corresponding nodes in the service model 
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instance 40 are computed. By default, the state of a node in instance may be extracted by the configurator to merge test 

the service model instance is determined based on the states information relevant to particular elements with default 

of all of the measurements associated with a node and the algorithm descriptions in order to generate a measurement 

states of all of its dependencies (children nodes) in the a S ent specification 46 for the ISP of interest, 
service model instance. The service model creation engine 5 

38 may support additional state computation policies. For TABLE 5 

example, a "measurementsOnly" policy indicates that only [io.i74.i73.23-fakeisp.net:DNS] 

the states of the measurements associated with a node must ipAddress-io.i74.i73.23 

be taken into account in determining the state of that node. hostNamc=dn S ii a kcis P .nct 

° 10 domai.nName~fakeisp.net 

Regarding attribute value settings, each service model category-ExtemalDnsServer 

node may derive attributes from the discovered instance 36 [10.174.173.23:SM-Host] 

* .... 4 r « . J1t , . . , measuiemeats^TCP-CoQnectiori-Ratc (10.174.173.23), 

to which it refers. The service model template syntax also (10 .n4.i73.23), IFstat (10.174.173.23) 

allows for hierarchical aggregation of attributes. This is ipAddrcss-io.i 74.1 73.23 

demonstrated in the append construct used for defining 15 hostName-dnsiAtcisp.net 

category attributes for the service model nodes. ^tegory- 6 

[10.137.196.52:SM-Host] 

SERVICE MODEL CREATION ENGINE ineasuiements-TCP-Cotinection-Rate (10.137.196.52), 

VMstat (10.137.196.52), IFstat (10.137.196.52) 

™ , . t . - Q . t . ipAddress=10.137.196.52 

The service model creation engine 38 incorporates the 20 hostName-maafes2i.fakeisp.net 

logic for processing a service model template 34 with the state-Alive 

discovered instance 38 to generate the service model category- 

instance 40. There are alternatives with regard to the order ^uVemen^TCPX ^nection-Rate(io.i37.i96.54), 

in which the engine 38 processes the service model template. VMstat(io.i37.i96.54), iFstat(io.i37.i96.54) 

In a sequential processing approach, it is assumed that the 25 ipAddress=io.i37.i96.54 

service model template was constructed such that the sec- ^^-mai^.fakeisp.net 

tions of the service model template are in an order in which category- 

they need to be processed. The engine can then process the [10.137.196.56:SM-Host] 

sections sequentially. The sequential processing enables ^^^m^^ 

simplification of the service model creation engine. ipAddress=io.i37. 196.56 

However, this approach burdens the template developer with hostName=mailfes23.fakeisp.net 

a requirement of manually determining the placement of state-Alive 

each section in the template based upon the order of pro- I?!f?°!7" „ „ , 

. y ■ ♦ • H ♦ - * .u [10.137.196.58:SM-Host] 

cessing. Moreover, since processing typically starts from the m e aS urement s -'rcp-Coanecaon-Rate(io.i37.i96.5S), 
host nodes, this approach may result in a number of service VMstat(io.i37.i96.58), iF B tat(i0.i37.i96.58) 
model host nodes that do not have additional nodes above ipAddress-io.i37.i96.58 
them when a service model instance graph is generated in a ^^ www ' fakeisp Qet 
manner to be described below. To avoid such "orphaned" category- 
nodes, the created service model instance must be further [io.i37.i96.69:SM-Hostl 

processed. 40 measwements=TCP-Connection-Rate(10. 137.196. 69), 

VMstat(10. 137.196.69), IFstat(10. 137.196.69) 

In the alternative hierarchical processing, a service model ipAddress=l0.137.l96.69 

creation engine 38 can use the "components" specifications hostName-wwwxyz.com. fakeisp.net 

in the different sections of the service model template 34 to ^te^ory^ 

determine the order for processing the sections of the 45 [io.i37.i96.70:SM-Host] 

template. Since the components list specifies the dependen- measurements-TCP-Connection-Rate(i0.i37.i96.70), 

cies of a node, before processing a section, all of the sections . VMstat(io.i37.i96.70), lFstat(io. 137.196.70) 

corresponding to each of the nodes types in the components t^^^^^^MaSBp^ 

list must be processed. Based on this rule, sections of the state=AKve 

templates which have no specified components are pro- 50 category- 

cessed first. Although this hierarchical approach requires 

more complexity than the sequential approach, it does not 

result in any orphaned nodes in the service model instance CUSTOMIZING THE SERVICE MODEL TO AN 

40. ISP'S ORGANIZATIONAL STRUCTURE 

55 As previously noted, the service model instance 40 of 

SERVICE MODEL INSTANCE SPECIFICATION FIG. 4 is customized to an ISP's organizational structure, so 

that ISP operations personnel only view the status of ser- 

Table 5 is an example of a portion of a service model vices and service elements that are of relevance to the 

instance specification for the service model template of personnel. A straightforward approach to customizing the 

Table 4 and the discovered instance of Table 3. The variables 60 service model instance to an ISP's organizational structure 

of the table are consistent with the variables of Tables 2-4. is to edit the service model template and explicitly include 

Referring now to FIG. 3, the service model instance 40 may nodes that capture the organizational dependencies. For 

be used by the view generator 42 to provide any of the three example, the nodes may be grouped according to categories 

views. Preferably, the service model instance is represented (e.g., InternalWeb services and Web Hosting services). Each 

as a graph of the nodes and edges that identify dependencies 65 of these categories may be managed by different operations 

among the nodes. The service model instance is also used by personnel. To accommodate this case, the service model 

the measurement agent configurator. Information from the template could be modified to define nodes that represent the 
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individual categories and dependencies that indicate the 
components of the different categories. Since the organiza- 
tional structure varies from one ISP to another, the approach 
would require that each ISP edit the service model templates 
to match their organizational structure. Editing the service 
model template to define the categories and the components 
relationships can be a tedious task, especially for a large ISP. 

An alternative approach is to allow an ISP to specify its 
organizational structure using the configuration interface 60 
previously described with reference to FIG. 4. The service 
model template 34 is pre-specified to exploit the configu- 
ration specification and to generate a service model instance 
that is customized to each ISP. The main advantage of this 
approach is that the ISP operator only has to primarily edit 
the configuration specification, which is much less complex 
than editing the service model template 34. 

The application of this less complex approach can be 
described with reference to Tables 1-5. Through the con- 
figuration interface 60, an ISP operator specifies ways in 
which the discovered services and service models are to be 
categorized in the discovered instance 36. In Table 1, the 
configuration specification indicates that the ISP uses the 
naming pattern *.com.fakeisp.net to identify web servers 
that are hosted for the businesses. Web servers that do not 
match this pattern are internal web servers that are used by 
the ISP's residential customers. Each of the discovery mod- 
ules 50, 52 and 54 then uses the configuratioa specification 
to determine a categorization of the services and service 
models that are discovered. A discovery module is also 
included in the discovery template 48 to discover and report 
on all the categories that have been discovered in the ISP's 
system. This information is used by the service model 
creation engine 38 to construct a service model instance 40 
that represents the ISP's organizational structure. To enable 
this, the service model template of Table 4 has a section that 
generates a node in the service model instance for each 
category in the discovered instance. The components list in 
this section maps all the services and service elements that 
are associated with a category to the corresponding node in 
the service model instance. Thus, by merely specifying the 
categorization of services and service elements using the 
configuration specification, an ISP operator can derive a 
service model instance that is customized for the ISP. 

EXAMPLE: READ MAIL SERVICE MODEL 

As an example implementation of the system of FIG. 3, 
the email service of "Read Mail" will be considered. The 
Read Mail service refers to a subscriber accessing his/her 
mailbox from the mail system of the ISR FIG. 5 illustrates 
a service topology for this service. Using a client application 
that supports the Post Office Protocol-Version 3 (POP3), a 
subscriber at a desktop computer 62 attempts to access mail. 
Internal to the ISP system, the request from the subscriber's 
computer 62 may be received and processed by one of many 
servers 64, 66 and 68 that constitute a mail service group 70. 
The servers within the group are front-end servers (FESs). 

Before the subscriber can access the appropriate mailbox, 
the mail server 64-68 that handles the request contacts an 
authentication server 72 to verify the identity of the sub- 
scriber. Typically, password identification is used in the 
Read Mail service. A subscriber database 74 is accessed in 
this process. Following the authentication process, the mail 
FES 64-68 accesses a mailbox 78 of the subscriber from a 
back-end content server 76 using the NFS service. The 
retrieved mail messages are transmitted to the computer 62 
of the subscriber. 
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There are several active and passive measurements that 
can be made to assess the health of the different elements 
involved in supporting the Read Mail service. A measure- 
ment system (MS) may be installed in the server farm of the 

5 ISP to perform measurements using agents executing on the 
MS and on the different ISP hosts. The different measure- 
ments that characterize the Read Mail service include an 
active service quality measurement of availability and 
response time made from the MS in the service farm, an 

10 active measurement of network throughput from the MS in 
the service farm to the POP sites, passive measurements of 
CPU and memory utilization passive measurements of TCP 
connection traffic and packet traffic to the mail servers 
obtained from agents executing on the mail servers, and 

1S passive measurements of NFS statistics (e.g., number of 
calls, timeouts, and duplicate transmissions) on the mail 
servers and the mail content servers. The active measure- 
ments attempt to assess the service quality as viewed from 
subscribers connecting to the POP sites, while the passive 

20 measurements may be used to assess resource utilization and 
traffic statistics that are critical for problem diagnosis. 

FIG. 6 is an illustration of an example of a view that may 
be presented to an ISP operator using the view generator 42 
of FIG. 3. While the oval-shaped nodes in the service model 

25 graph represent the different services and service elements, 
the arrows represent measurements of services and service 
elements. The root of the service model graph is the Read 
Mail service, represented by oval 80. The state of this node 
represents the overall health of the Read Mail service, as 

30 assessed by the MS located in the service farm of the ISP. 
That is, the overall health is assessed without considering the 
state of the network links from the server farm to the POP 
sites. In one embodiment, the overall health is represented 
by color coding the oval 80. For example, oval 80 may be 

35 shaded green to designate a positive health of the Read Mail 
service, and may be shaded red if the Read Mail service has 
degraded in its availability or performance. 

Typically, there is no direct measure of the overall health 
of the Read Mail service. Instead, the state of the service 

40 must be inferred, based on the states of the different mail 
FESs 64-68 that together enable the Read Mail service. 
Direct active measures of availability and performance, and 
passive measurements of TCP statistics to the POP3 service 
port, together contribute to the determination of the status of 

45 the Read Mail service. The active and passive measurements 
are performed at each of the mail FESs, as indicated by the 
arrows corresponding to the second level service oval 82 in 
FIG. 6. 

The next level of the service model graph reflects the 

50 dependencies of the Read Mail service on one element and 
two services. Oval 84 is the dependency of the Read Mail 
service on a POP3 server executing on the mail FES. Oval 
86 represents the authentication service for verifying the 
identity of the subscriber from which a Read Mail request is 

55 received. Oval 88 represents the NFS service used by the 
particular mail FES. Considering the POP3 server 84 first, 
the health of the server is measured based on the ability to 
establish a TCP connection as part of the active Read Mail 
service quality measurement and the time required to estab- 

60 lish the connection. In turn, the health of the POP3 server 
may be impacted by the link, represented by oval 90, 
interconnecting the mail FES to the measurement station 
(from which the active test is run) and the health of the mail 
FES host, represented by oval 92. As shown in FIG. 6, both 

65 the link 90 and the host 92 include four performance 
parameters that are measurable in determining the health of 
the nodes. While not shown in FIG. 6, the state of the various 
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nodes may be represented by color coding or by other expert who is responsible for the mail service and the mail 

display means for distinguishing the states of the nodes. servers uses the service model depicted in FIG. 6, since the 

Regarding the dependency of the authentication service expert is mainly interested in the states of the email services 

86 on the mail service 82, since it is possible that the and servers. The authentication and NFS services are 

authentication service is healthy but a specific mail FES is 5 included in the service model representation, since these 

failing to perform authentication, the authentication service services can adversely impact the Read Mail service. In 

is first represented from the point of view of each of the mail contrast, the links between the service farm and the POP 

FESs 94. Direct measures of the authentication delay when sites are not included in the model, since they do not affect 

accessing through a mail FES are used to determine the state me Read service from the perspective of the email 

of the mail authentication service 86 in relation to that mail 10 expert. 

FES 94. The service model for an authentication service w _ A nwmn „™ rt ™ 

node, whose state affects the state of the mail FES-specific ^^SpTiT^^ 

authentication node, is not expanded in the service model REPRESENTATION 

graph of FIG. 6. Likewise, the NFS service 88 is not shown can De seen in FIGS. 5 and 6, the service model maps 

as being expanded in the service model graph. The service 15 different measurements for some of the nodes in the service 

model dependency is handled in much the same way as the model graphs. The node to which a measurement maps 

authentication service dependency. depends on the semantics of the measurement (i.e., which 

As previously noted, the service model graph of FIG. 6 logic node or nodes are targeted by the measurement) and 

represents the Read Mail service in isolation. To represent the location or locations from which the measurement is 

the end-to-end service, a service model must take into 20 made. In the simplest case, each measurement directly maps 

account the state of the DNS service used by subscribers to to one of the nodes in the service model. In some cases, 

resolve the ISP mail domain to each one of the mail FESs, measurements may span multiple service elements and there 

and the state of the network links between the different POP may not be a direct mapping of a measurement to a node in 

sites and the ISP server farm. Clearly, since the different the service model. In such cases, composite measurements, 

POP sites use different routes to connect to the service farm, 25 which are combinations of measurements being made in the 

a subscriber's perception of the end-to-end service may vary, ISP system, may have to be composed and mapped to the 

depending upon the POP site that the subscriber is using. nodes in the service model. For example, suppose Link (x,y) 

FIG. 7 is a service model graph for the Read Mail service as is a network link interconnecting hosts x and y in an ISP 

perceived by a subscriber connected to the ISP system via system, and suppose Link (x,y) is comprised of Link (x,z) 

POP m . The Read Mail POP m service is represented by oval 30 and l ink (z,y). If measurements are made from x, Link (x,y) 

96 and has the Read Mail service 80 of FIG. 6 as one of its and Link (x,z) can be directly measured. The status of Link 

dependencies. However, the service 80 is not expanded. (z,y) has to be derived from the status of Link (x,y) and Link 

While not shown in FIG. 7, the graphing preferably includes (x,z). 

color coding or other designation for nodes, such as the Each of me nodes ^ me ^ice model has an instanta- 

service 80, which are not fully expanded. neous state associated with it. As previously noted, the state 

The other dependencies on the Read Mail POP„ service of a node reflects the health of the service or service element 

96 include the link 98 and the DNS service 100. The health that it represents. A number of policies can be used to 

of the link is determined by measurements of the perfor- compute the state of service model nodes. One policy 

mance parameters throughput, packet loss, delay and con- ^ computes the state of a node based on all of its measure- 

nectivity. The health of the DNS service 100 is determined ments. Another policy assigns weights to the different mea- 

by measurements of availability and performance. The DNS surements based on an estimate of their reliability, as gauged 

service 100 is shown as having the dependency POP m DNS by a domain expert. Yet another policy determines the state 

server 102. In turn, the server 102 has two dependencies, of a node based on its measurements in combination with the 

namely POP m DNS host 104 and the link 106. 45 states of its dependencies in the service model graph. The 

There are several ISP management functions thai can states of the measurements associated with a node may be 

exploit the capabilities of the service models. For example, determined by applying baselining and thresholding tech- 

a service model for operational monitoring may be config- niques to the measurement results, 
ured to indicate the status of the different elements providing 

a service. When a failure occurs, the service model indicates 50 DISCOVERY METHODOLOGIES 

which element or elements have been affected by the failure. Returning to FIGS. 2 and 4, the preferred embodiment of 

Moreover, by traversing the service model graph top-down, me management system 56 includes a discovery engine 58 

an operator can determine the root-cause of the failure. For mal automatically discovers services, service elements and 

example, in FIG. 6, when a problem is noticed with the dependencies among services and service elements. FIGS. 7 

overall Read Mail service 80, an operator can traverse the 55 anc j g illustrate applicable discovery methodologies. Similar 

service model graph to determine whether the problem is to existing management system implementations, a first 

caused by a specific mail FES or whether all of the mail phase of discovery is performed from a management station 

FESs are experiencing a problem. Assuming a similar jo^ wn ich may be external to the service farm of the ISP 

scenario, moving down the service model graph, the opera- system. Predominantly, this phase involves active tests that 

tor can further determine whether the problem is related to 60 generate test traffic and query all of the ISP hosts to detect 

authentication failure, NFS failure, or a failure of the POP the existence of different types of servers 12, 14, 22, 26 and 

application server 84. 28. That is, the phase detects execution dependencies, as 

Since services and service elements can be organized defined above. Component and organizational dependencies 

based on domains of responsibility in a service model, ISP are also detected during this phase. Moreover, some of the 

operations personnel need only monitor and diagnose the 65 inter-service dependencies are discovered, but the second 

services that fall in their domain of responsibility. In the phase is focused on the inter-service dependencies, since it 

Read Mail service example of FIG. 5, an email operations may not be possible to discover all the inter-service depen- 
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deocies that exist using tests executed from outside the ISP to access hosts using their host names, rather than their more 

host. The second phase uses an internal view of the ISP difficult to remember IP addresses, DNS stores the host 

system. Preferably, the two phases are executed sequentially, name-to-IP address mapping for all of the hosts in the ISP 

with the second phase utilizing the discovered information system. Moreover, the exchange of email messages across 

output by the first phase to direct its operations. Different 5 hosts occurs using the mail exchange records maintained by 

mechanisms can be employed in the internal discovery ^ DNS. Name server (NS) records in the DNS database 

phase. Both phases of discovery must be executed ^ t0 identify authoritative name service for the ISP 

periodically, so as to discover new services and service domain— these are name servers that arc externally acces- 

elements that may have been introduced into the ISP envi- sible from tne S lobal Internet and are the authorities that are 

ronment. 10 contacted when users in the global Internet attempt to access 

t TTisi o - j**u£* any hosts in the ISP system. Moreover, service groups, such 

In FIG. 8, a single discovery agent 110 is used in the first 3 . J ' . . 6 j 

, * rp, i j riii as an email service group, are enabled via round-robin 

phase external discovery process. The solid line 112 repre- , , u • • i ♦ a • *u tm^o r 

* *l J* * * *• r^xro + a * scheduling mechanisms implemented m the DNS servers. In 

sents the discovery agent contacting the DNS server 14 to ~« , . v 4 . . , u , ? . t 

t 4 c , t . r Tort 4 te ™ , , , , summary, the domain name system holds a wealth of lnfor- 

get a list of hosts in the ISP system. The dashed lines from 4 . v 4 - . 4 . , c \ A - * Tori 

5. 4llfl . ; . ^-**u- rnation that is critical for auto-discovery of ISP services, 

the discovery agent 110 indicate active tests being executed 15 u t , . J 4 , 

. ,< j. jo o However, additional mechanisms are necessary to com pie - 

by the discovery agent to detect various types of dependen- 4 „ ' , , , . J r 

cies that exist m DNS-based discovery mechanisms. 

,„«.".„ . e , i i . * One of the first steps in discovery is to determine all of the 

FIG. 9 is an illustration of the second phase interna hosts ^ exist m the Isp m Most existm netWQrk 

discovery process to th* phase a number of internal managcmcnt systcms have takcn onc of ^ approaches. A 

discovery agents 114, 116, 118 and 120 are utilized. THe first J ' h \ s to ^ m address ^ is cither 

solid fines having arrows at one end indicate the discovery dfied b afl tQr Qr fa determined bascd on thc local 

of mter-service dependencies. The dashed lines mdicate the subnet of the measurement host . Th e address range is 

flow of discovered instance information back to the man- scumd using ping tQ responses from all IP . eri abled 

agement station 108. ^ hosts ^ approac h jg to use the default router 

Both phases of the discovery process may employ one or configured for a host to boot-strap the discovery. Commu- 

more different basic categories of discovery techniques. nication using SNMP with the router and using its routing 

Under the service-independent techniques, hosts and net- tables may be used to discover hosts in the local subnet, 

working equipment (routers 20, hubs 18, and POP sites 16) Also, routing table information may be used to discover 

which are not specific to any service are discovered. As a 3Q ome r routers that can provide additional host information, 

second category, service-generic techniques (which may be ^ alternative approach that is complementary to either of 

the same techniques, but with appropriate parameterization me ^ traditional approaches is to obtain a list of all of the 

for each service) may be used to discover instances of hosts in me ISP system using information available with the 

different services. In order to do so, typically, such discovery domain name seryice From thc host name of the managc _ 

techniques exploit common characteristics of different ser- 35 ment stat i on m wn i ca the discovery process executes, the 

vices to discover instances of the services. An example of Isp domain name can be deduced. Using the default name 

this second category is a discovery technique for News, Web configured for the management system, the discov- 

and email services. Since all of these services rely upon the ery process que ries DNS to obtain the NS records that list 

same transport protocol (i.e, TCP), a discovery technique me authoritative name servers for the ISP domain. One or 

can discover the existence of application servers by con- ^ more of these name serV ers are contacted by the discovery 

necting to the TCP ports corresponding to the services. process t0 obtain a ^ of named hosts m ±t Isp system 

Another category of techniques may be referred to as the Zone transfer capabilities supported by all DNS servers are 

service-specific-but-application-independent techniques. used for this purpose, as is known in the art. While ISPs 

Techniques in this category are specific to the service. They usually manage a single domain, some ISPs may have 

are intended to monitor, but may be used for discovery that 45 multiple domains under their control. To discover all of the 

is independent of the specific application server that is being hosts that exist in the different domains, the discovery 

used to implement the service. For example, the discovery of process must be informed of the existence of all the different 

the relationship between the services offered by POP3 email domain names. This information cannot be automatically 

servers and the service provided by SMTP-based email deduced by the discovery process. 

servers is possible using application-independent 50 The steps that are executed in the first phase of discovery 

techniques, since the relationship is accessible from the are identified in FIG. 10. It is not critical that the steps be 

domain name service in the form of mail exchange (MX) followed in the order shown in the figure. Following the step 

records. 122 0 £ discovering the hosts, the application servers are 

A fourth category may be referred to as application- discovered in step 124. The existence of application servers 

specific techniques. Many inter-service dependencies may 55 of different types (e.g., Web, Email, News, FTP, DNS, NFS, 

need to be discovered in a manner that is very specific to the and Radius) is verified by active tests that emulate typical 

application servers providing the services. For example, to client requests directed at the different TCP and UDP ports 

discover the dependency of the web service on NFS, the corresponding to the different service types. A response to an 

discovery technique may have to query the configuration file emulated request has to be interpreted in a service-specific 

in the web application server that is providing the web 60 manner. By observing the header in a response from the web 

service. Since the format and contents of a web application service, the discovery process determines the type of web 

server's configuration file are specific to the application, the server that is executing on the host machine. Likewise, by 

discovery technique is application -specific. processing the response returned by the email and News 

F.RST PHASE EXTERNAL DISCOVERY „ ^t^^^S^t^ 

An often under utilized component of an ISP system is the phase of discovery. For instance, discovery agents installed 

domain name service (DNS). In order to allow subscribers on the ISP host machines in the second phase of discovery 
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may process a web application server's configuration files to 
discover NFS dependencies that the server may have. Since 
web server configuration files are typically specific to the 
type of server, the server type information provided by the 
first phase of discovery can be used to determine the 
processing capabilities of the discovery agent or agents that 
must be deployed in the second phase. The server type 
information may also be used to determine specific mea- 
surements which must be targeted for the server to monitor 
its status. 

In step 126, the existence of Web, Email, News, DNS and 
other service groups has to be determined using the DNS. By 
querying the DNS, the discovery process determines a list of 
domain names that have multiple IP addresses associated 
with them. For each name in the list, the discovery process 
then determines whether each of its IP addresses hosts a 
common application server. If so, the name likely represents 
a DNS round-robin service group that supports a common 
service. For example, suppose that all of the IP addresses 
corresponding to the name www.fakeisp.net host web serv- 
ers. In this case, www.fakeisp.net represents a web service 
group. Note that in this process, a host that has two network 
interfaces, and therefore is assigned to different IP addresses, 
may be listed as a service group. Using the virtual host/ 
server discovery heuristics discussed below, all such hosts 
can be removed from the service group list. 

In step 128, the MX records for the ISP domain are 
accessed from the DNS system. The MX records indicate the 
mail servers that must be contacted to deliver mail to the ISP 
domain. The list of SMTP-based mail servers thus deter- 
mined represent the servers that handle delivery of incoming 
mail to the subscribers of the ISP. Discovery of these servers 
is essential to automatically generating a service model for 
the email service that represents the delivery of mail from 
the Internet to the subscribers. Moreover, the mail gateways 
may be managed by a different entity than the one that 
manages mail servers that are internal to the ISP. 

One of the critical measures of the performance of an 
email service of an ISP is the round-trip delay between the 
transmission of an email message from a source host and its 
reception at the intended destination. This measurement can 
be used to assess email delivery times in the ISP domain, 
from the ISP domain to locations on the Internet, and from 
locations on the Internet to the ISP domain. Since the email 
service uses different protocols and, hence, different appli- 
cation servers to send mail and to receive mail, in order to 
initiate round-trip delay measurements of email, it is essen- 
tial to determine relationships between the different types of 
email servers on the ISP domain (e.g., which SMTP server 
can be used to send mail to a POP3/IMAP-based server). 
This discovery is executed at step 130. Since mail forward- 
ing is predominantly based on the MX records maintained in 
the DNS database, by querying the DNS system for MX 
records corresponding to each of the POP3/IMAP servers, 
the discovery process determines the mail service relation- 
ships in the ISP domain. 

Various approaches can be adopted to discover the ter- 
minal servers that exist in an ISP POP site in step 132. The 
most straightforward approach uses SNMP queries to obtain 
the MIB-II system description from all the hosts in the ISP 
network. Based on the replies, the discovery process can 
identify the hosts that are terminal servers. An alternative 
approach is based on the observation that because they need 
to operate and manage thousands of terminal servers, most 
ISPs have specific naming conventions that they use when 
naming their terminal servers. In fact, the naming conven- 
tion more often indicates the association between terminal 
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servers and POP sites, so that when a problem is reported by 
a subscriber using a POP site, the ISP operations staff can 
quickly decide which of the terminal servers needs to be 
checked in order to diagnose the problem. With this 

5 approach, an ISP provides a regular expression representing 
the naming convention used as input to the discovery 
process. By matching the list of hosts that are discovered 
with the naming convention, the discovery process not only 
determines the terminal servers that exist, but also deter- 

10 mines the POP site to which a terminal server is assigned. 
Another key advantage of this approach is that it performs 
discovery without generating any additional network traffic. 

The approach of exploiting name conventions may also be 
used in step 134 to categorize the other services of the ISP. 

15 As an example, for each web site that it hosts for its business 
customers, a particular ISP may assign internal names of the 
form *. com.fakeisp.net, so that a hosted web site named 
www.customer-domain.com will have a corresponding entry 
for www.customer-domain.com.fakeisp.net in the internal 

20 DNS database of the ISR As in the case of terminal servers, 
by permitting an ISP to specify its naming conventions, the 
discovery process composes a categorization of services that 
is customized to the target ISP system. This categorization 
can be based on geographical locations of services, based on 

25 business relationships, and/or based on the delegation of 
responsibilities among operators. The categorization infor- 
mation can be used to automatically define the customized 
service model for each ISP, with special nodes in the service 
model representing a collection of nodes pertaining to the 

30 same category. 

SECOND PHASE DISCOVERY 

By treating the ISP system as a "black box," the first phase 

35 of external discovery detects most of the execution, com- 
ponent and organizational dependencies of the ISP. 
Additionally, some of the inter-service dependencies are 
discovered. The internal discovery process is focused solely 
on detecting inter-service dependencies, particularly those 

4Q that are not discovered by taking an external viewpoint. For 
example, the relationship between a mail server and an NFS 
server is not discoverable from an external viewpoint. 

There are two basic approaches for conducting the inter- 
nal discovery. One approach uses network probes, while the 

45 other approach uses special-purpose discovery agents. 
Regarding the first approach, software probes installed at 
strategic locations on the network can snoop on packet 
transmissions. Since most TCP/IP communication is based 
on source/destiny port numbers, by processing the headers 

50 of packets that are captured by the probe, a software probe 
can deduce many of the relationships that exist among 
services. For example, a UDP packet transmitted from a mail 
server to the NFS port of an NFS server indicates that the 
mail server depends on the NFS server for its content 

5S storage. 

An advantage of the approach that utilizes network probes 
is that the approach enables discovery of inter-service 
dependencies independent of the specific types of applica- 
tion servers residing on the host. Moreover, since it relies on 

60 just the ability to capture packets on the wire, this approach 
handles UDP and TCP : based services equally well. 

The key difference between the approach of using net- 
work probes and the approach of using special-purpose 
discovery agents is that unlike the network probes, the 

65 discovery agents do not snoop on packet transmissions. 
Instead, the discovery agents use a number of operating 
systems and application-specific mechanisms to discover 
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inter-service dependencies. These mechanisms include (1) Advantages of using discovery agents, as compared to 

processing service configuration information and (2) network probes, include the reduction of overhead on the 

application-dependent monitoring tools. Referring first to ISP hosts, the relaxation of security concerns, and the fact 

the processing service configuration information, applica- that all of the discovery agents do not need to be deployed 

tion servers determine their dependencies on other services 5 at the same time. Instead, the deployment of discovery 

from one or more configuration files. By processing the agents can occur at the discretion of the ISP. As and when 

content of the configuration files, discovery agents can new discovery agents are installed on the ISP hosts, addi- 

discover inter-service dependencies. An example of this is tional information is discovered about the ISP system, 

the processing of the web server's configuration file to FIG. 11 is a process flow of steps relevant to the second 

discover whether it has a dependency on an NFS service. 10 phase of the discovery process. In steps 136 and 137, the 

While processing the web server's configuration file, the information that is obtained in the first phase of discovery is 

discovery agent can also determine if the same application usc d to generate an incomplete service model instance. As 

is being used to host multiple "virtual" servers (which is previously noted, the first phase of discovery provides the 

commonly used by ISPs to host web sites on behalf of their necessary information for identifying component 

business customers). Typically, web server configuration 15 dependencies, organizational dependencies, execution 

files are specific to the type of server executed on the web dependencies and some of the inter-service dependencies. A 

server in use. The server type determination performed first instance generator matches the service model template 

during external discovery (i.e., the first phase of discovery) with the auto-discovered information from the first phase to 

is used for deciding the location and format of the configu- generate the incomplete service model instance. However, 

ration files. 20 other inter-service dependencies are not discoverable using 

While many Unix operating systems use configuration the techniques of the first phase (e.g., the * relationship 

files that are defined in an application-specific manner, between a mail server and an NFS server). 

Windows NT-based systems store all application configura- f Q s t ep s 138 and 139, the holes in the incomplete service 

tion information in the registry. In the Windows NT systems, mo del instance are identified and information obtained in the- 

while the registry can be processed in an application- 25 first phase is used to determine appropriate discovery 

independent manner, the specific configuration attributes actions. The incomplete service model instance is used to 

have to be interpreted in an application-specific manner. determine the types of relationships that must be examined. 

Thus far, only forward-looking discovery agents have In the mail service example, if a host is discovered in the first 
been identified. These are agents that discover dependencies phase to be running a POP3 server, the second phase may be 
of a service on other services by querying configuration files 30 used to discover the name file service and authentication 
of the application providing the service. Sometimes it is service used by the POP3 server on the particular host, 
easier to implement backward-looking discovery agents to i n step 140, the network probes and/or the special- 
discover the dependencies on a service (i.e., discover which purpose discovery agents are deployed in a manner deter- 
other services are using the service). For example, the m ^ cd during execution of the step 139. For example, 
configuration file of the mail authentication server may application-specific knowledge may be used to parse con- 
indicate which of the mail servers are depending on the figuration files or log files, or may be used to search a 
authentication server. One of the ways of implementing configuration registry for a particular server instance execut- 
backward-looking discovery agents is by processing appli- mg on a particular host. The network probes and/or discov- 
cation server configuration files. ^ er y agents generate service dependency outputs in step 141. 

Turning now to the second mechanism of using The outputs are used in a second instance generation to 

application-independent monitoring tools, this approach is complete the service model instance in step 143. 
particularly attractive for services that communicate using 

TCP. The netstat utility can be used to determine the TCP EXTENSIBLE DISCOVERY ARCHITECTURE 

connections that exist on an ISP host. A discovery agent that 4S Siace new ne twork services and service elements are 

executes this tool can periodically discover information Deing deployed at a rapid pace, it is important that the 

about the source and destination ports and the host locations discovery methodologies be implemented in an extensible 

for TCP connections, which can then be used to deduce man ner, allowing new discovery capabilities to be incre- 

inter-service dependencies. mentally added to the management system. FIG. 12 depicts 

This second approach of using application-independent 59 the extensible architecture for discovery components previ- 

monitoring tools exploits the fact that most TCP implemen- ously described with reference to FIG. 4. The discovery 

tations enforce a three-minute delay for connections in the modules 50, 52 and 54 represent the logic used for discovery 

TIME_WAIT state of TCP, so that a connection persists for of different services and service elements. The discovery 

about three minutes even after it is no longer in use. template 48 is the key to the extensibility of the auto- 

Consequently, whenever the discovery agent is executed, it 55 discovery architecture in the sense that it drives how dis- 

is likely to detect all the TCP connections that may have covery is performed. The template defines the different 

been established in the three minutes prior its execution. services and service elements that need to be discovered, and 

This same approach does not work for UDP-based services, the specific discovery modules that can be used to discover 

since UDP is connectionless, and there is no state that is these elements. The template also establishes the format of 

maintained at either the source or the destination. 60 me outputs from the modules. 

The approach of monitoring TCP connections can be used The discovery engine 58 drives the auto-discovery pro- 

to discover dependencies such as those that exist between cess. The discovery engine interprets the discovery template 

mail servers and mail authentication servers, between serv- 48 and for each of the service or service element types 

ers and back-end databases, between Radius/TACACS specified in the template, the engine invokes the correspond- 

authentication servers and terminal servers, and between 65 ing discovery module 50, 52 and 54 specified in the tem- 

similar relationships. Again, discovery agents can be plate. All of the discovery modules report the results of their 

forward-looking or backward-looking. execution back to the discovery engine. The discovery 
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template contains instructions for the discovery engine to tion pertains. There are three types of instructions that are 

process the output of the discovery modules and to record specified in the configuration file. All of these instructions 

them as a discovered instance 36. specify regular expression patterns that must be applied 

Some discovery modules may rely on the discovery against the IP address or host name of the service or service 

results of other discovery modules. For example, a DNS 5 element. The instructions in the configuration file are (1) 

round-robin service group discovery module for web ser- criteria that instruct the discovery modules to include or 

vices relies on identifying which hosts support web services, exclude specific services or service elements, (2) criteria that 

which is an output of the web service discovery module 50. instruct the discovery modules to associate specific services 

To accommodate these relationships, as part of the interface or service elements with certain categories, and (3) criteria 

that the discovery engine 58 exposes to the discovery 10 for discovering terminal servers and for extracting POP 

modules, the engine provides ways for accessing and search- site-to-terminal server mapping from the terminal server 

ing the instances discovered by other discovery modules. names. 

In contrast to the discovery modules 50, 52 and 54, the The second function of the query processor 148 is to 

discovery engine 58 is designed to be independent of the provide the discovery modules 146 with access to previously 

services that need to be discovered. Consequently, to dis- 15 discovered instances. Based on configuration and discovered 

cover new services or service elements, a user merely has to instance information obtained from the query processor, the 

provide a discovery template specification or one or more discovery modules perform tests on the ISP system and 

discovery modules for each new element. By providing an re P ort meir discovery output to the discovery engine. A 

alternate discovery module for a service that is already discovery instance generator module 150 of the discovery 

supported, a user can also enhance capabilities of an existing 20 engine processes the results of the discovery modules and 

discovery system. outputs the discovery instance in an appropriate format. An 

In practice, there are two significantly different of ^ ch * [ ormat was Piously set forth in Table 

approaches to designing the discovery engine 58 and the 3 - ™* formate of the discovery template and the discovery 

discovery modules 50-54. A first approach is to enable the* mstance m thereb y hldden from the dlscoverv moduIes - 

discovery engine to control the discovery process. The As previously noted, the second approach to designing the 

discovery engine accesses the discovery template and deter- discovery engine 58 and the discovery modules 50-54 of 

mines the order in which sections of the template are FIG. 12 is to establish an arrangement in which the discov- 

processed. On the other hand, in the second approach, the ery process is driven by the modules. In this alternative 

discovery modules drive the discovery process. In effect, embodiment, the discovery engine processes the template 

this is an event-driven approach, since the results obtained 30 once, invoking the discovery modules simultaneously. From 

from one module will trigger subsequent activities by other this point, the discovery modules determine when different 

modules. elements in the ISP system are discovered. The discovery 

Regarding the first approach in which the discovery modules execute periodically, looking for new instances, 

process is driven by the discovery engine 58, FIG. 13 3S Some discovery modules are independent in the sense that 

illustrates the logical building blocks of the discovery me y m DOt reUant on other modules for discovery. These 

engine. The discovery engine executes periodically and each independent modules begin executing immediately, 

time it starts, the engine processes the discovery template. As and when a discovery module 50-54 discovers a new 

By considering the values of the dependencies variable instance, the discovery module forwards its results to the 
for each of the sections in the discovery template, the 40 discovery engine 58. Based on the dependencies on a 
discovery engine determines the order in which the sections discovery module, as specified in the discovery template 48, 
must be processed. Thus, the discovery engine includes a ^ en B ine 58 forwards the results to other discovery mod- 
template parser 142. Sections of the template which have no ules for which ^ results are relevant. The availability of 
dependencies are processed first. A module loader 144 new results ( e *S-> me discovery of a new host) may trigger 
* directs the relevant information to the appropriate discovery 45 discovery by other modules (e.g., the web server module 
module 146 for processing a particular section in which no checks to determine if a web server is executing on the new 
dependencies are identified. After all such sections are host )> and this process continues. A key advantage to this 
processed, the discovery engine iterates through the list of approach, as compared to the engine-driven discovery 
template sections, choosing sections which have not been approach, is that multiple discovery modules may be execut- 
processed and which have their dependencies determined by 50 m S ^ P araUel > discovering the ISP's services. In this 
earlier processing. This process is repeated periodically to approach, the discovery engine 58 mainly functions as a 
discover new instances as and when they are added to the facilitator of communication among the discovery modules, 
system being discovered. In one application, the discovery A variant of this approach may not even involve the dis- 
engine uses the exec system call to invoke the discovery covery engine, with the discovery modules registering inter- 
modules at separate processes. By doing so, the discovery 5S cst in otDCr discovery modules and information concerning 
engine is able to handle discovery modules written in a newl y discovered instances being directly communicated 
variety programming environments. amoa g Ae discovery modules, without involving a discov- 

A query processor 148 of the discovery engine 58 per- eFV en S me - 
forms two functions. First, when a module 146 is activated, INTEGRATING DISCOVERY WITH SERVICE 
the processor 148 queries the discovery engine to obtain 60 MODELS 
configuration information that guides the discovery mod- 
ules. In FIG. 4, the configuration information is generated In the scenario in which the management system uses 
from the configuration interface 60 that is manipulableby a service models for management of Internet services, there 
user. Table 1 was previously included to depict a typical are two ways in which discovery can be integrated with 
configuration file. Each line in the file represents an instruc- 65 service models. In a looser integration, the output of dis- 
tion for one of the discovery modules. The first column of covery (the discovered instance) is integrated with a service 
the line identifies the discovery module to which the instruc- model template that oudines the structure of a service, and 
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the integration automatically generates a service model dencies among said discovery modules with respect to 

instance that is customized for the ISP system being man- sequencing deployment of said discovery modules in imple- 

aged. However, the preferred integration is one that provides menting said automated techniques, 

a tighter integration, and involves driving auto-discovery 6. The method of claim 4 wherein said step of forming 

and service model instantiation from a common template. In 5 said discovery template includes providing instructions 

this preferred approach, for each node in the service model, regarding organizing outputs of each said discovery module 

corresponding discovery template specifications are pro- for display during said step of displaying discovered 

vided. The discovery and service model-specific compo- instances, said step of displaying being executable in the 

nents of the template can either be processed in a single absence of said discovery template, 

application or can be processed separately. This approach 10 7. The method of claim 1 wherein said formatting of said 

towards tighter integration of discovery and service model discovery template includes storing said discovery template 

templates is attractive for several reasons. Firstly, the service in a format compatible with a discovery engine for driving 

model template can serve to constrain the discovery process, said automated techniques. 

since only services and service elements that are specified in 8. The method of claim 1 wherein said formatting of said 

the service model template need to be discovered. Secondly, 15 discovery template includes arranging said instructions in an 

depending upon its design, the service model template could order which determines a sequence for implementing said 

end up using some of the outputs of the discovery process. automated techniques, said step of displaying including 

Using a common template permits tighter syntax checking employing a service model template to determine a display 

across the discovery and service model components of a format. 

template. Thirdly, the two-phase approach to discovery 20 9. The method of claim 1 further comprising a step of 

described above fits in well with the service model concept. enabling cooperation between memory for storing said dis- 

The inter-service dependencies that need to be discovered in covery template and software for implementing said auto- 

the second phase (internal discovery) can be determined mated techniques such that said software determines a 

based on the service model template. Finally, the discovery sequence for implementing said automated techniques, said 

process itself can be determined based on the service model 25 step of displaying including employing a service model 

template specification. The discovery process may attempt template to determine a display format, 

to traverse the service model template tree from a root node 10. A method of modeling a selected service available via 

down. At each level, it attempts to discover all services or a network comprising steps of: 

service elements of the types specified by the node, provid- enabling discovery capabilities for executing diverse dis- 

ing all of the children of a node have been discovered. If this 30 covery routines specific to acquiring decentralized 

is not the case, the discovery process proceeds to first information that identifies services and service ele- 

discover all instances of the children nodes. Continuing the ments of said network; 

tree traversal recursively, the discovery process discovers all forming a discovery template that is specific to said 

instances that are necessary to build the service model for selected service, including providing data relevant to 

the ISP system being managed. 35 triggering selected said discovery routines for acquir- 

What is claimed is: ing said decentralized information, said selected dis- 

1. A method of modeling services available via networks covery routines being related to acquiring instance 
comprising steps of: information identifying services and service elements 

selecting a core service of interest; accessed in providing said selected service; 

forming a discovery template that is specific to said core 40 storing said discovery template; and 

service such that said discovery template includes activating an automated process for collecting said decen- 

instructions for implementing automated techniques for tralized information so as to drive modeling of said 

discovering instances of services and service elements selected service, including accessing said discovery 

which are anticipated as being cooperative in execution template to initiate said selected discovery routines, 

of said core service within a network environment, 45 11. The method of claim 10 wherein said step of forming 

including formatting said discovery template for access said discovery template includes identifying types of service 

during implementation of said automated techniques; elements, including identifying at least one server type such 

and that said step of activating said automated process includes 

displaying discovered instances of said services and ser- transmitting queries that are specific to discovering instances 

vice elements in isolation from said discovery template. 50 of servers of said server type. 

2. The method of claim 1 wherein said step of forming 12. The method of claim 11 wherein said step of enabling 
said discovery template includes identifying types of said said discovery capabilities includes providing software dis- 
service elements without specifying actual service elements, covery modules specific to an Internet Services Provider 
said step further including identifying types of services (ISP) system, said discovery template being formed without 
which are anticipated as being cooperative in execution of 55 being specific to said ISP system, thereby enabling use of 
said core service. said discovery template with any one of a plurality of ISP 

3. The method of claim 2 wherein said step of identifying systems. 

said types of service elements includes specifying types of 13. The method of claim 10 wherein said step of forming 

servers and wherein identifying said types of services said discovery template includes providing instructions for 

includes specifying services enabled by said types of serv- 60 configuring outputs generated in response to said discovery 

ers. routines, said modeling being performed without entering 

4. The method of claim 2 wherein said step of forming said outputs into said discovery template. 

said discovery template includes associating each identified 14. The method of claim 10 wherein said step of forming 

type of service and type of service element with a discovery said discovery template includes forming template sections 

module for implementing said automated techniques. 65 such that each template section (1) is specific to a type of 

5. The method of claim 4 wherein said step of forming service element, (2) specifies, at least one said discovery 
said discovery template further includes specifying depen- routine for identifying a service element of said type, and (3) 
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specifies dependencies that said at least one discovery indicative of selected services which are anticipated as being 

routine has on outputs of other said discovery routines. cooperative in execution of said core service, said stored 

15. A system for modeling a selected core service avail- discovery template having data indicative of said at least 
able via a network comprising: some discovery module means. 

a plurality of discovery module means for accessing 5 17. The system of claim 16 further comprising a service 

decentralized information available on said network model instance engine means for generating a graphical 

and for generating outputs indicative of service ele- representation of said core service in response to said 

ments of said network; outputs of said discovery module means, said graphical 

a discovery engine means connected to each of said representation including nodes representing said service 

discovery module means, said discovery module means 10 elements that are indicated within said outputs of said 

b being responsive to said discovery engine means with selected discovery module means, said service model engine 

respect to invoking individual discovery module means being independent of said discovery template, 

means; and \g The system of claim 16 wherein said discovery 

a stored discovery template that is specific to said core 15 template further includes information indicative of opera- 
service to be modeled, said discovery template includ- tional dependencies among said discovery module means in 
ing data indicative of selected said service elements discovering said selected services and selected service ele- 
which are anticipated as being cooperative in execution ments. 

of said core service, said discovery template further 19, The system of claim 18 wherein said discovery 

including data indicative of selected discovery module 2Q template has an arrangement that controls a sequence of 

means for identifying said selected service elements, invoking said selected discovery module means, said 

said discovery engine means being connected to access arrangement being at least partially determined by said 

said discovery template such that said discovery engine dependencies. 

means automatically invokes said selected discovery 20. The system of claim 18 wherein said discovery engine 
module means; 25 means determines a sequence of invoking said selected 
wherein said discovery template controls said discovery discovery module means, said sequence being at least par- 
engine means and wherein said discovery engine means tially determined by said dependencies, 
controls said discovery module means, said outputs that 21. The system of claim 18 wherein said discovery 
are indicative of said service elements being recorded module means cooperate to determine a sequence of gener- 
for representation of said core service independently of 30 ating said outputs, said sequence being at least partially 
said discovery template. determined by said dependencies. 

16. The system of claim 15 wherein at least some of said 

discovery module means are configured to generate outputs ***** 
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